ESQL - match operator included in non-snapshot builds #116819

carlosdelest · 2024-11-14T14:34:33Z

Removes snapshot restriction for the match operator (:), allowing it to be used on serverless and regular builds.

Add documentation for the match operator:

Adding docs involved some changes, as Match is both a function and an operator:

Changing AbstractFunctionTestCase so a new search operators category is added for operators.
Add an almost empty MatchOperatorTestCase to make sure the SVG signature is generated for the new operator.

github-actions · 2024-11-14T14:34:50Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-11-14T16:20:35Z

Hi @carlosdelest, I've created a changelog YAML for you.

elasticsearchmachine · 2024-11-14T16:23:35Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2024-11-14T16:23:35Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

fang-xing-esql

Thank you @carlosdelest, I added some comments to the generated docs.

fang-xing-esql · 2024-11-18T17:43:00Z

docs/reference/esql/functions/kibana/definition/count.json

  "examples" : [
    "FROM employees\n| STATS COUNT(height)",
-    "FROM employees \n| STATS count = COUNT(*) BY languages \n| SORT languages DESC",
+    "FROM employees\n| STATS count = COUNT(*) BY languages\n| SORT languages DESC",


This seems to be a leftover of a prior PR, not relevant to this PR. Just want to mention it in case it is included by mistake.

This was re-generated by tests - happy not to include it, but shouldn't we use the opportunity to keep these generated files up to date?

Same happened with qstr, where examples were generated as uppercase instead.

Yeah, I think it is good to keep the generated docs up to date, it makes main clean.

fang-xing-esql · 2024-11-18T17:45:36Z

docs/reference/esql/functions/types/match_operator.asciidoc

+[%header.monospaced.styled,format=dsv,separator=|]
+|===
+field | query | result
+keyword | keyword | boolean


The types listed here seem to be different from the match function's types - docs/reference/esql/functions/types/match.asciidoc, should we make them consistent?

fang-xing-esql · 2024-11-18T18:05:02Z

docs/reference/esql/functions/search.asciidoc

+
+Supported types:
+
+include::types/match.asciidoc[]


It will be nice to include an example of the match operator here.

I think for doing that it would be needed to add a new MatchOperator class with the corresponding @FunctionInfo annotation that contains the examples, so they are auto-generated and can be linked.

We could create that class just for doc purposes and avoid instantiating it, or make it subclass Match - it could even be used to represent the operator instead of using Match for both the function and the operator.

WDYT?

The search.asciidoc is not a generated file, we can just add an example of match operator manually without adding a new class. in.asciidoc has an example of including a csvtest there. Perhaps just pick one example with tag from match-operator.csv-spec and add it here.

Nice! I didn't know we could add CSV tests directly without generating the examples via tests. Thanks!

Added an example in 22f4388, now search operators looks like:

fang-xing-esql · 2024-11-18T18:07:57Z

docs/reference/esql/functions/search.asciidoc

+[.text-center]
+image::esql/functions/signature/match_operator.svg[Embedded,opts=inline]
+
+Supported types:


This is a duplicated line in the doc, the link below also has supported types before the table.

Thanks for the catch! Fixed in af0b035

fang-xing-esql · 2024-11-18T18:12:04Z

docs/reference/esql/functions/kibana/definition/match_operator.json

+      "params" : [
+        {
+          "name" : "field",
+          "type" : "keyword",


docs/reference/esql/functions/kibana/definition/match.json has both keyword and text types, should we make them consistent(needs additional tests on text type for the operator)?

You're right - I added the missing field types in 68907ff.

needs additional tests on text type for the operator

I think text type is already covered in CSV tests? Are there other tests you would expect to check this?

I mean to add text type in the MatchOperatorTests so that the text type can be included in the docs automatically, which you already did in the latest commit.

Just for completeness, in docs/reference/esql/functions/kibana/definition/match.json, we have 4 combinations:

keyword, keyword
keyword, text
text, keyword
text, text

docs/reference/esql/functions/kibana/definition/match_operator.json has 2 combinations:
keyword, keyword
text, text

Adding two additional tests for the other two combinations in MatchOperatorTests should be able to make the types of : the same as match function.

Got it - adding them in cdf896c

fang-xing-esql

LGTM, thank you @carlosdelest, as far as CI is happy ! I added a minor comment on the types for kibana.

fang-xing-esql · 2024-11-19T18:50:29Z

docs/reference/esql/functions/kibana/definition/match_operator.json

+      "params" : [
+        {
+          "name" : "field",
+          "type" : "keyword",


Just for completeness, in docs/reference/esql/functions/kibana/definition/match.json, we have 4 combinations:

keyword, keyword
keyword, text
text, keyword
text, text

docs/reference/esql/functions/kibana/definition/match_operator.json has 2 combinations:
keyword, keyword
text, text

Adding two additional tests for the other two combinations in MatchOperatorTests should be able to make the types of : the same as match function.

# Conflicts: # x-pack/plugin/esql/src/main/antlr/EsqlBaseLexer.tokens # x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.tokens # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseLexer.interp # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseLexer.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java

carlosdelest · 2024-11-20T14:19:09Z

@elasticmachine run elasticsearch-ci/bwc-snapshots

carlosdelest · 2024-11-20T17:06:27Z

@elasticmachine run elasticsearch-ci/8.17.0 / bwc-snapshots

elasticsearchmachine · 2024-11-21T06:46:34Z

💔 Backport failed

Status	Branch	Result
✅	8.x
❌	8.18	The branch "8.18" is invalid or doesn't exist

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 116819

(cherry picked from commit ea4b41f)

carlosdelest · 2024-11-21T07:25:39Z

💚 All backports created successfully

Status	Branch	Result
✅	8.17

Questions ?

Please refer to the Backport tool documentation

)

) (cherry picked from commit ea4b41f)

carlosdelest added 4 commits November 14, 2024 14:37

Add code to generate docs for match operator

6b3d019

Add docs

9425de5

Fix other docs (caps and spaces) after regenerating all

78998e7

Remove snapshot from match operator capability

084b4ed

carlosdelest added >feature auto-backport Automatically create backport pull requests when merged v9.0.0 v8.17.0 labels Nov 14, 2024

carlosdelest added 3 commits November 14, 2024 15:37

Fix Railroad diagram labels

fe147f3

Spotless

a1e799a

Fixed signature SVG

019312f

carlosdelest added test-release Trigger CI checks against release build Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch :Search Relevance/Search Catch all for Search Relevance labels Nov 14, 2024

carlosdelest and others added 2 commits November 14, 2024 17:20

Update docs/changelog/116819.yaml

733e7bd

Updated changelog

6a32cea

carlosdelest marked this pull request as ready for review November 14, 2024 16:22

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels Nov 14, 2024

carlosdelest requested a review from fang-xing-esql November 14, 2024 16:23

carlosdelest requested review from ChrisHegarty, afoucret, ioanatia and tteofili November 14, 2024 16:24

carlosdelest added 2 commits November 18, 2024 11:07

Remove lexer / parser isDevVersion

c350403

Add missing generated files

881d59c

Merge branch 'main' into feature/esql-match-operator-non-snapshot

9731ea4

fang-xing-esql reviewed Nov 18, 2024

View reviewed changes

carlosdelest added 4 commits November 19, 2024 09:17

Remove duplicate line

af0b035

Merge branch 'main' into feature/esql-match-operator-non-snapshot

83b0c9a

Add example to match operator

22f4388

Add missing type "text" for the operator

68907ff

carlosdelest requested a review from fang-xing-esql November 19, 2024 17:24

fang-xing-esql approved these changes Nov 19, 2024

View reviewed changes

carlosdelest added 3 commits November 20, 2024 08:11

Add missing type combinations for the operator to docs

cdf896c

Regenerate lexer / parser after merge

673e5d9

Merge branch 'main' into feature/esql-match-operator-non-snapshot

cf80243

elasticsearchmachine added v8.18.0 and removed v8.17.0 labels Nov 20, 2024

carlosdelest added the v8.17.0 label Nov 21, 2024

carlosdelest merged commit ea4b41f into elastic:main Nov 21, 2024
16 of 17 checks passed

carlosdelest mentioned this pull request Nov 21, 2024

[8.x] ESQL - match operator included in non-snapshot builds (#116819) #117224

Merged

carlosdelest added a commit to carlosdelest/elasticsearch that referenced this pull request Nov 21, 2024

ESQL - match operator included in non-snapshot builds (elastic#116819)

0625db4

elasticsearchmachine added the backport pending label Nov 21, 2024

carlosdelest added a commit to carlosdelest/elasticsearch that referenced this pull request Nov 21, 2024

ESQL - match operator included in non-snapshot builds (elastic#116819)

97165df

(cherry picked from commit ea4b41f)

carlosdelest mentioned this pull request Nov 21, 2024

[8.17] ESQL - match operator included in non-snapshot builds (#116819) #117227

Merged

carlosdelest added a commit that referenced this pull request Nov 21, 2024

ESQL - match operator included in non-snapshot builds (#116819) (#117224

5a9c05e

)

elasticsearchmachine pushed a commit that referenced this pull request Nov 21, 2024

ESQL - match operator included in non-snapshot builds (#116819) (#117227

0304b92

) (cherry picked from commit ea4b41f)

carlosdelest mentioned this pull request Nov 22, 2024

ES|QL MATCH roadmap #116261

Closed

alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this pull request Nov 28, 2024

ESQL - match operator included in non-snapshot builds (elastic#116819)

32798c5


		Supported types:

		include::types/match.asciidoc[]

ESQL - match operator included in non-snapshot builds #116819

ESQL - match operator included in non-snapshot builds #116819

Uh oh!

Conversation

carlosdelest commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 14, 2024

Uh oh!

elasticsearchmachine commented Nov 14, 2024

Uh oh!

elasticsearchmachine commented Nov 14, 2024

Uh oh!

elasticsearchmachine commented Nov 14, 2024

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Nov 20, 2024

Uh oh!

carlosdelest commented Nov 20, 2024

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 21, 2024

💔 Backport failed

Uh oh!

carlosdelest commented Nov 21, 2024

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

carlosdelest commented Nov 14, 2024 •

edited

Loading

fang-xing-esql Nov 19, 2024 •

edited

Loading

carlosdelest Nov 19, 2024 •

edited

Loading

carlosdelest Nov 19, 2024 •

edited

Loading